|Poster:||TheXmanCometh||Date:||Aug 24, 2015 2:51pm|
|Forum:||faqs||Subject:||Web History for a website with no content|
About a year ago the site went away but you can try to access it (internal server error.) This happens even if you try to reach a page that you know existed:
When I search here I receive "Page cannot be crawled or displayed due to robots.txt" and I think this is happening only because robots.txt cannot be accessed. Is there any way you can allow searches for sites like this that are alive but all content was removed? You should be able to rely on the cached version of the file to verify that the site owner wasn't preventing site searches.